Technological progress in the field of optical image acquisition allows the use of camera-based driver assistance systems which are located behind the windshield and capture the area in front of the vehicle in the way the driver perceives it. The functionality of these assistance systems ranges from automatic headlights to the detection and display of speed limits, lane departure warnings, and imminent collision warnings.
Starting from just picking up the area in front of the vehicle to a full 360° panoramic view, cameras can be found in various applications and different functions for driver assistance systems in modern vehicles. It is the primary task of digital camera image processing as an independent source of sensor data or in conjunction with laser or lidar sensor data to detect, classify, and track objects in the image area. Classic objects typically include various vehicles such as cars, trucks, two-wheel vehicles, or pedestrians. In addition, cameras detect traffic signs, lane markings, guardrails, free spaces, or other generic objects.
Automatic learning and detection of object categories and their instances is one of the most important tasks of digital image processing and represents the current state of the art.
Modern driver assistance systems use various sensors including video cameras to capture the area in front of the car as accurately and robustly as possible. This environmental information, together with driving dynamics information from the vehicle (e.g. from inertia sensors) provide a good impression of the current driving state of the vehicle and the entire driving situation. This information can be used to derive the criticality of driving situations and to initiate the respective driver information/alerts or driving dynamic interventions through the brake and steering system.
Since the actually available friction coefficient or equivalent information about the current road condition is typically not provided or cannot be measured or determined in driver assistance systems that are ready for series production, the times for issuing an alert or for intervention are in principle determined based on a dry road with a high adhesion coefficient between the tire and the road surface.
This results in the following fundamental problem. Accident-preventing or at least impact-weakening systems warn the driver or intervene so late that accidents are prevented or accident impacts acceptably weakened only if the road is really dry. But the effect of driver dynamic interventions via the brake and steering system is critically dependent on the friction coefficient of the ground. Moisture, snow, and ice reduce the coefficient of friction available between the tire and the road considerably compared to a dry road. If the road provides less adhesion due to moisture, snow, or even ice, an accident can no longer be prevented and the reduction of the impact of the accident does not have the desired effect.
A known approach to counteracting this fundamental problem is to evaluate camera images for the purpose of estimating road conditions and for deriving estimated friction coefficients.
Document DE 10 2004 018 088 A1 discloses a road recognition system having a temperature sensor, an ultrasound sensor, and a camera. The temperature, roughness, and image data (road data) obtained from the sensors is filtered and compared to reference data, and a margin of safety is created for the comparison. The condition of the road surface is determined based on the comparison of the filtered road data with the reference data. The road surface (e.g. concrete, asphalt, dirt, grass, sand, or gravel) and its condition (e.g. dry, icy, snow-covered, wet) can be classified in this way.
Document WO 2012/110030 A2 discloses a method and apparatus for estimating the friction coefficient using a 3D camera, such as a stereo camera. At least one image of the vehicle environment is taken using the 3D camera. A height profile of the road surface is created in the entire area in front of the vehicle from the image data of the 3-D camera. The anticipated local coefficient of friction of the road surface in the area in front of the vehicle is estimated from the height profile. In individual cases, classification of the road surface as snow cover or a muddy dirt road can be based on specially determined elevation profiles.
However, the known methods make high demands on the required sensors. The methods or apparatuses mentioned require a temperature and an ultrasound sensor in addition to a camera, or a camera that is configured as a 3D sensor, to obtain sufficiently robust classification results.